Traffic light detecting system and method

ABSTRACT

Systems and methods are provided for determining the status of traffic lights through a mobile device and communicating the status to the user of the device. The mobile device detects its geographical location, direction and elevation. The mobile device also receives information related to traffic lights, such as location and type. The device calculates positional deviation from the device to the traffic light and generates one or more prompts in the form of audible or tactile cues that progressively guide the user to point the device at the traffic light. The mobile device detects the image and color of the traffic light and determines the illumination and sign status of the traffic light. Then the mobile device communicates the traffic light status to the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of the filing date of U.S.Provisional Patent Application No. 61/297,455 filed Jan. 22, 2010, thedisclosure of which is hereby incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to traffic light detectionsystems and methods. More particularly, the present invention relates toidentifying the state of traffic lights using a mobile device.

2. Description of Related Art

Blind travelers normally use the sound of traffic to judge whether atraffic light is green when deciding to cross a street. Building devicesthat can read traffic lights for one who cannot see is a technicalchallenge. Such electronic aids that can reliably announce the state ofthe traffic light typically require special-purpose hardware to beinstalled on traffic lights, and therefore such devices are oftenprohibitively expensive in the past.

Technologies that do not require the installation of special hardware ontraffic lights have been used in custom built vehicles, e.g., the robotcars seen in the DARPA challenge that sense their environment whennegotiating the city streets. However, such technologies typically relyon sensors. Such sensor-based solutions are likely to remain intractableon mainstream mobile devices. Consequently such solutions remain out ofreach of the average consumer because of the high cost.

SUMMARY OF THE INVENTION

In one embodiment, a method of identifying a traffic light status isprovided. The traffic light status comprises at least one of a colorillumination status and a sign status of a traffic light. The methodcomprises receiving, at a user device, geolocation data associated withthe traffic light and the user device. The user device geolocation dataincludes a location of the user device, and the traffic lightgeolocation data includes a location of the traffic light. The methodalso includes detecting an elevation and a direction of the user device;and determining the status of the traffic light with the user devicebased on the geolocation data associated with the traffic light and theuser device, the elevation and the direction associated with the userdevice. The method also includes communicating the status to a user ofthe user device.

In accordance with another embodiment, a device for identifying atraffic light status is provided. The traffic light status comprises atleast one of a color illumination status and a sign status of a trafficlight. The device comprises a memory for storing information, includinggeolocation data associated with a traffic light and the device,direction and elevation information of the device. The devicegeolocation data includes a location of the device, and the trafficlight geolocation data includes a location of the traffic light. Theprocessor is coupled to the memory. The processor is configured toreceive geolocation data associated with the traffic light and thedevice, and to detect an elevation and a direction associated with thedevice. The processor is also configured to determine the status of thetraffic light with the device based on the received geolocation data,the detected elevation and the direction of the device. The processor isfurther configured to communicate the status to a user of the device.

In accordance with a further embodiment, a method for identifying atraffic light status with a server computer is provided. The trafficlight status comprises at least one of a color illumination status and asign status of a traffic light. The method comprises receiving, from auser device, geolocation data associated with the traffic light and theuser device, elevation and direction data associated with the userdevice. The user device geolocation data includes a location of the userdevice, and the traffic light geolocation data includes a location ofthe traffic light. The method also comprises determining the status ofthe traffic light based on the received geolocation data, the elevationand direction data, and transmitting the status to the user device.Determining the status of the traffic light comprises generating one ormore instructions to orient the user device to face the traffic lightand transmitting the instructions to the user device. The method furthercomprises instructing the user device to communicate the status to theuser.

In accordance with a further embodiment, a server apparatus is employedto identify a traffic light status. The traffic light status comprisesat least one of a color illumination status and a sign status of atraffic light. The apparatus comprises a memory for storing information,including geolocation data associated with a traffic light and a userdevice, direction and elevation information of the user device. The userdevice geolocation data includes a location of the device, and thetraffic light geolocation data includes a location of the traffic light.The processor is coupled to the memory. The processor is configured toreceive geolocation data associated with the traffic light and the userdevice, and to detect an elevation and a direction associated with theuser device. The processor is also configured to determine the status ofthe traffic light with the device based on the received geolocationdata, the detected elevation and the direction of the device.Determining the status of the traffic light comprises generating one ormore instructions to orient the user device to face the traffic lightand transmitting the instructions to the user device. The processor isfurther configured to instruct the user device to communicate the statusto the user.

In accordance with a further embodiment, a system is provided. Thesystem comprises memory means for storing information data. Theinformation data includes geolocation data associated with a trafficlight and a user device, direction and elevation information of the userdevice. The user device geolocation data includes a location of thedevice, and the traffic light geolocation data includes a location ofthe traffic light. The system also includes means for detecting alocation of a traffic light and a location of the user device, means fordetecting a direction of the user device, means for detecting anelevation of the user device, and means for detecting a light signal ofthe traffic light. The system further includes means for capturing animage of the traffic light. The system also includes processor means fordetermining a status of the traffic light. The traffic light statuscomprises at least one of a color illumination status and a sign statusof a traffic light. The processor means is also for generating one ormore instructions to orient the user device to face the traffic lightbased on the geolocation data associated with the traffic light and theuser device, the elevation and the direction of the user device. Thesystem further includes means for outputting the traffic light status toa user of the user device.

It is to be appreciated that, unless explicitly stated to the contrary,any feature in any embodiment, alternative or example can be used in anyother embodiment, alternative or example herein and hereafter.

In one example, determining the status of the traffic light comprisesdetermining, from the geolocation data of the user device, if a currentlocation of the user device is adjacent to an intersection. In the casewhere the current location is adjacent to the intersection, the methodfurther comprises prompting the user to orient the user device to facethe traffic light.

In one alternative, determining the status of the traffic light with theuser device includes generating one or more instructions based on thegeolocation data of the traffic light and the user device, the elevationand the direction of the user device. Orienting the user device to facethe traffic light also includes providing the instructions to the user.

In another alternative, generating the one or more instructions toorient the user device includes generating a map for an area between theuser device and the traffic light; and calculating deviations from theuser device to the traffic light based on the map.

In a further alternative, the method comprises capturing one or moreimages of the traffic light and a surrounding area by an image capturedevice of the user device. In this case, determining the status of thetraffic light is performed based on the captured images.

In another example, the method includes receiving information related tothe traffic light, where the information comprises at least one of asize, a type and a timing sequence of the traffic light. In thissituation, determining the status of the traffic light is performedbased on the received information related to the traffic light.

In one alternative, the information related to the traffic light isreceived from a server computer.

In another alternative, the information related to the traffic light isgenerated based on the captured images.

In yet another example, determining the status of the traffic light withthe user device comprises receiving light signals from a plurality oflight sources, where one of the light sources comprises the trafficlight. Determining the status of the traffic light also includesfiltering light signals for frequency ranges emitted by the trafficlight.

In another example, determining the status of the traffic light includesrecognizing signs associated with the traffic light.

In yet another example, the method includes receiving audibleinformation related to the status of the traffic light, and determiningthe status of the traffic light is further based on the audibleinformation.

In one alternative, communicating the traffic light status to the userincludes generating an audible output.

In another alternative, communicating the traffic light status to theuser includes generating a tactile output.

In one example, the location of the traffic light and the location ofthe user device is detected by a geographical position device.

In another example, the direction associated with the user device isdetected by a digital compass.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial diagram of a system in accordance with aspects ofthe invention.

FIG. 2A is a pictorial diagram of a system in accordance with aspects ofthe invention.

FIG. 2B is a functional diagram in accordance with aspects of theinvention.

FIG. 3 is a flowchart in accordance with aspects of the invention.

FIG. 4 is a functional diagram in accordance with aspects of theinvention.

FIG. 5A is an exemplary diagram in accordance with aspects of theinvention.

FIG. 5B is an exemplary diagram in accordance with aspects of theinvention.

FIG. 6 is an exemplary diagram in accordance with aspects of theinvention.

DETAILED DESCRIPTION

Aspects, features and advantages of the invention will be appreciatedwhen considered with reference to the following description of exemplaryembodiments and accompanying figures. The same reference numbers indifferent drawings may identify the same or similar elements.Furthermore, the following description is not limiting; the scope of theinvention is defined by the appended claims and equivalents.

In accordance with aspects of the invention, a system determines thestatus of traffic lights through a mobile device and describes thestatus to the user of the device. The mobile device detects itsgeographical location, e.g., through a GPS system and determines if theuser is at an intersection or is otherwise near a traffic light based onthe knowledge from a map database. The mobile device also receives thegeographical location and other information related to the traffic lightat the intersection from a database storing traffic lights information.The direction and elevation of the device is calculated (e.g., by themobile device). The device then generates one or more prompts such as inthe form of audible or tactile cues that progressively guide the useruntil the device is precisely pointed at the light. The mobile devicedetects the image and color of the traffic light. The detected image andcolor are processed, and the status of the traffic light is determined.Then the mobile device communicates the traffic light status to theuser.

As shown in FIG. 1, a system 100 in accordance with one aspect of theinvention includes a server computer 110, a mobile device 160, a network90, a traffic light 102 and a satellite 103. The mobile device 160 isconnected through network 90 to the server 110. The mobile device mayhave a built-in GPS receiver to receive geolocation data from satellite103. The traffic light 102 may have lights 102 a-102 c, each of which isdedicated to one corresponding street lane. The lights emitted by thetraffic light 103 may be detected by the mobile device 160 and befurther processed by the device or the server or both. As theconfiguration 200 of FIG. 2A shows, the network 90 may connect with oneor more mobile devices 160 and 170, server computers 110 and 112 and aplurality of databases 136, 138, 140 and 142. Various types of data,such as user-related information, traffic light information,location/map data, image processing programs, may be stored in thesedatabases and downloaded to the server or the mobile device. Variousfunctions, such as image processing, may be performed on the mobiledevice 160 or on the server 110.

As illustrated in the functional diagram of FIG. 2B, the server computercontains a processor 120, memory 130 and other components typicallypresent in general purpose computers. The memory 130 stores informationaccessible by processor 120, including instructions 132 and data 134that may be executed or otherwise used by the processor 120. The memory130 may be of any type capable of storing information accessible by theprocessor, including a computer-readable medium, or other medium thatstores data that may be read with the aid of an electronic device, suchas a hard-drive, memory card, ROM, RAM, DVD or other optical disks, aswell as other write-capable and read-only memories. Systems and methodsmay include different combinations of the foregoing, whereby differentportions of the instructions and data are stored on different types ofmedia.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computer codeon the computer-readable medium. In that regard, the terms“instructions” and “programs” may be used interchangeably herein. Theinstructions may be stored in object code format for direct processingby the processor, or in any other computer language including scripts orcollections of independent source code modules that are interpreted ondemand or compiled in advance. Functions, methods and routines of theinstructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. For instance, although the systemand method is not limited by any particular data structure, the data maybe stored in computer registers, in a relational database as a tablehaving a plurality of different fields and records, XML documents orflat files. The data may also be formatted in any computer-readableformat. By further way of example only, image data may be stored asbitmaps comprised of grids of pixels that are stored in accordance withformats that are compressed or uncompressed, lossless (e.g., BMP) orlossy (e.g., JPEG), and bitmap or vector-based (e.g., SVG), as well ascomputer instructions for drawing graphics. The data may comprise anyinformation sufficient to identify the relevant information, such asnumbers, descriptive text, proprietary codes, references to data storedin other areas of the same memory or different memories (including othernetwork locations) or information that is used by a function tocalculate the relevant data.

The processor 120 may be any conventional processor, such as processorsfrom Intel Corporation or Advanced Micro Devices. Alternatively, theprocessor may be a dedicated device such as an ASIC. Although FIG. 2functionally illustrates the processor and memory as being within thesame block, it will be understood by those of ordinary skill in the artthat the processor and memory may actually comprise multiple processorsand memories that may or may not be stored within the same physicalhousing. For example, memory may be a hard drive or other storage medialocated in a server farm of a data center. Accordingly, references to aprocessor or computer will be understood to include references to acollection of processors or computers or memories that may or may notoperate in parallel.

The server 110 may be at one node of network 90 and capable of directlyand indirectly communicating with other nodes of the network. Forexample, server 110 may comprise a web server that is capable ofcommunicating with user devices 160 and 170 via network 90 such thatserver 110 uses network 90 to transmit and display information to auser, such as person 191 or 192 of FIG. 1B, on display of client device160. Server 110 may also comprise a plurality of computers that exchangeinformation with different nodes of a network for the purpose ofreceiving, processing and transmitting data to the client devices. Inthis instance, the user devices will typically be at different nodes ofthe network than any of the computers comprising server 110.

Network 90, and intervening nodes between server 110 and user devices,may comprise various configurations and use various protocols includingthe Internet, World Wide Web, intranets, virtual private networks, localEthernet networks, private networks using communication protocolsproprietary to one or more companies, cellular and wireless networks(e.g., WiFi), instant messaging, HTTP and SMTP, and various combinationsof the foregoing. Although only a few computers are depicted in FIG.1-2, it should be appreciated that a typical system can include a largenumber of connected computers.

The user devices 160 and 170 may comprise mobile devices capable ofwirelessly exchanging data with a server over a network such as theInternet. By way of example only, user device 170 may be awireless-enabled PDA or a cellular phone capable of obtaininginformation via the Internet. The user may input information using asmall keyboard (in the case of a Blackberry phone), a keypad (in thecase of a typical cell phone) or a touch screen (in the case of a PDA).

Each user device may be configured with a processor 120, memory 130 andinstructions 132. Each client device 160 or 170 may be a device intendedfor use by a person 191-192, and have all of the components normallyused in connection with a mobile device such as a central processingunit (CPU), memory (e.g., RAM and internal hard drives) storing data andinstructions such as a web browser, an electronic display 162 (e.g., asmall LCD touch-screen or any other electrical device that is operableto display information), and user input 163 (e.g., keyboard,touch-screen and/or microphone), a network interface device (e.g.,transceiver and antenna), as well as all of the components used forconnecting these elements to one another. The output components on eachuser device may include a speaker 168 and a tactile output 166.

Memory 132 in each user device may store data 134 such as computer codethat, in response to the detected light, orientation and position of thedevice, generates a set of prompts that continuously guide the user toorient the device to the traffic light. Data 134 may also include animage processing library 142 that consists of image recognition routinesand appropriately tuned image filters to detect traffic lights. Historyof intersections and traffic lights data may be recorded in memory 132.

The user devices may also include one or more geographic positioncomponents to determine the geographic location and orientation of thedevice. For example, client device 160 may include a GPS 174 receiver todetermine the device's latitude, longitude and/or altitude position. Thegeographic position components may also comprise software fordetermining the position of the device based on other signals receivedat the client device 160, such as signals received at the antenna fromone or more cellular towers or WiFi access points. It may also includean accelerometer, gyroscope or other acceleration device 172 todetermine the direction in which the device is oriented. By way ofexample only, the acceleration device may determine its pitch, yaw orroll (or changes thereto) relative to the direction of gravity or aplane perpendicular thereto. In that regard, it will be understood thata client device's provision of location and orientation data as setforth herein may be provided automatically to the user, to the server,or both.

Besides the GPS data receiver 174 and the accelerometer 172, each userdevice may also include other components that help to detect theposition, orientation and elevation of the device. Such componentsinclude but are not limited to, a digital compass 176, an inclinometer178 and an altimeter 186.

Each user device may include image and/or color capture components suchas a camera 184, one or more image sensors 180 and one or more imagefilters 182.

Although certain advantages are obtained when information is transmittedor received as noted above, aspects of the invention are not limited toany particular manner of transmission of information. For example, insome aspects, information may be sent via a medium such as an opticaldisk or portable drive. In other aspects, the information may betransmitted in a non-electronic format and manually entered into thesystem. Yet further, although some functions are indicated as takingplace on a server and others on a client, various aspects of the systemand method may be implemented by a single computer having a singleprocessor.

Server 110 may store map-related information 140, at least a portion ofwhich may be transmitted to a client device. For example and as shown inFIG. 2A, the server may store map tiles, where each tile comprises a mapimage of a particular geographic area. A single tile may cover an entireregion such as a state in relatively little detail and another tile maycover just a few streets in high detail. In that regard, a singlegeographic point may be associated with multiple tiles, and a tile maybe selected for transmission based on the desired level of zoom. The mapinformation is not limited to any particular format. For example, theimages may comprise street maps, satellite images, or a combination ofthese, and may be stored as vectors (particularly with respect to streetmaps), bitmaps (particularly with respect to satellite images), or flatfiles.

The various map tiles are each associated with geographical locations,such that the server 110 and/or client device are capable of selecting,retrieving, transmitting, or displaying one or more tiles in response toreceiving one or more geographical locations.

The system and method may process locations expressed in different ways,such as latitude/longitude positions, street addresses, streetintersections, an x-y coordinate with respect to the edges of a map(such as a pixel position when a user clicks on a map), names ofbuildings and landmarks, and other information in other referencesystems that is capable of identifying a geographic locations (e.g., lotand block numbers on survey maps). Moreover, a location may define arange of the foregoing. Locations may be further translated from onereference system to another. For example, the user device 160 may accessa geocoder to convert a location identified in accordance with onereference system (e.g., a street address such as “1600 AmphitheatreParkway, Mountain View, Calif.”) into a location identified inaccordance with another reference system (e.g., a latitude/longitudecoordinate such as (37.423021°, −122.083939°)). In that regard, it willbe understood that exchanging or processing locations expressed in onereference system, such as street addresses, may also be received orprocessed in other references systems as well.

FIG. 2B illustrates that information related to traffic light may bestored in the server 110. Such information includes but is not limitedto geographical location data, type and scale (e.g., size, shape andelevation) of the traffic light, pictures and other data related to thevisual or positional features of the traffic light. User-specific oruser-device specific data, such as history of the intersections that theuser has crossed, may also be stored in the server.

In one embodiment, the databases storing different types of data mayreside on the same server, as shown in the configuration 240 of FIG. 2B.For example, locations of traffic light may be integrated with the mapdata in server 110. In another embodiment, the databases may reside ondifferent servers distributed through the network, as illustrated inFIG. 2A. Data related to user information may be stored in database 136.Traffic light related information may reside on database 138.Location/map data may be stored in database 140. Database 142 maycontain calculation routines and modules for image processing.

FIG. 3 depicts a flowchart 300 of one embodiment of the invention. Itwill be understood that the operations do not have to be performed inthe precise order described below. Rather, various steps can be handledin a different order or simultaneously.

In one example, as shown in block 310, a user holding a mobile device,such as device 160, may walk along a busy urban area. The mobile devicedesirably continuously receives GPS data and detects its current GPScoordinates. The device may also detect positioning data such aselevation and orientation of the device.

Based on the detected current location and the knowledge of mapinformation from a map database, the mobile device is able to correctlydetermine if the user has come to an intersection in block 312. The mapinformation may be retrieved from the map database by server 110 whichsends the data to the mobile device. If the user's current location isat an intersection, the mobile device may prompt the user to stop andinvoke the function of querying the state of the intersection in block314. If the current location is not an intersection, the user devicedesirably gives no indication or prompt, and continues to detect thepresent location as the user continues walking.

In block 316, the user invokes the device to query the state of theintersection. The device makes a request to retrieve traffic lightinformation from a server connected to the device. The device thenreceives traffic light related information such as the location,geometrical data (e.g., size, elevation, shape, etc.) and other data inblock 318.

In an alternative, the mobile device may have prior knowledge of trafficlight information, which comprises geolocation of traffic lights. Theknowledge may be obtained from a server, such as server 110 or adatabase, such as database 138, on a real time basis, or may bedownloaded to the device. Therefore, the mobile device may automaticallydetermine that the user is at an intersection and there is a trafficlight at the intersection based on the present location, map data andthe traffic light information. In this scenario, the device may promptthe user of the existence of a traffic light.

If the device decides that there is no traffic light at thisintersection in block 320, it may communicate to the user about thestatus of the intersection in block 334.

If the device decides that there is traffic light at this intersection,it may proceed to detect the traffic light in block 322. The device maydetect light of varying wavelengths from the surrounding area and filterthe received wavelengths according to the wavelengths emitted from thetraffic light. The device may also capture an image of the traffic lightand the adjacent area and analyze the image to find the targeted trafficlight.

If the device can not detect the traffic light in block 324, the devicemay determine that the user is not pointing the device to the trafficlight. It then, in block 326, calculates a vector of positionaldeviations from the device to the traffic light based on the detectedorientation, elevation and position of the user device, as well as basedon the scale and location data of the traffic light. The device may alsoincorporate the detected information of the surrounding area in thecalculation.

The vector of deviations may comprise horizontal deviation and verticaldeviation. For example, in scenario 500 in FIG. 5A, the user standing atintersection 501 and holding device 160 may initially point the deviceat the direction of building 510, and therefore the device deviates fromthe desired traffic light 102 c by an angle α to the east. In thisscenario, the device may prompt the user to move the device to the leftby angle α. The device may also be configured to continuously prompt theuser to move the device by a fraction of angle α and prompt the user tostop moving the device until the desired position is reached. If theuser's move is larger than desired and points the device to thedirection of light 102 a instead of light 102 c, the device may promptthe user to move to the east by an angle β. In another scenario 505 ofFIG. 5B, the user device may be held too high by the user, and thuspoints to the building 515 over the traffic light 102 by an angle θ. Thedevice may prompt the user to move the device downward by angle θ. Thedevice may also be held too low and points to the lower portion of thetraffic light pole so the device may ask the user to move the deviceupward by angle μ.

The information related to the traffic light, such as scale andelevation may be obtained by the user device from a server databasestoring such data. If the user device cannot obtain this informationfrom such a database, it may acquire the information by capturing andprocessing an image of the traffic light.

Returning to FIG. 3, based on the detected deviation from the trafficlight, the user device may generate one or more simple instructionsprompting the user to move the device to minimize the deviation in block328. For example, the instructions may be output to the user in the formof spoken message, such as “point up and to the left.” The devicecontinues the loop of detecting the traffic light, calculating thedeviation and prompting the user to adjust the direction of the deviceuntil the device is precisely pointed at the light.

When the traffic light is detected, the device may filter and/or processthe color images captured by an image capture component, such as acamera, in block 330. The device then determines the status of thetraffic light accordingly. In block 332, the device informs the trafficlight status to the user, for example, through a speaker in spokenlanguage “green, pass.”

The device may communicate to the user through non-audible methods, suchas haptic output, the color of the traffic light and/or the status ofthe intersection. The device may be configured to provide the user withoptions to choose the type of communication.

Aspects of the invention will now be described in greater detail withregard to FIG. 4, which illustrates a system diagram of the embodimentsof the present invention. Here, user device 160 may contain a positionand direction detection module 408 that receives geolocation data 402.The position and direction module 408 comprises components such as a GPSreceiver 174, a digital compass 176, an altimeter 186 and aninclinometer 178.

The approximate location of the client's device may be found with anumber of different technologies. For example, server 110 may receivegeolocation information from the GPS receiver embedded in the userdevice. Thus the device may have access to latitude and longitudeinformation. This information may be received by server 110 duringconnection with the user device in conformance with communicationprotocols. For example, the device may use a system such as GoogleChrome or the browser of the Android operating system, each of which maybe configured with user permission to send GPS information to trustednetwork sites. Server 110 may use this information to determine alocation of the device. Because the accuracy of GPS determinations maydepend on the quality of the device and external factors such asenvironment, the device may further transmit data indicative ofaccuracy. For example, the user device 160 may inform the server 110that the transmitted latitude/longitude position is accurate within 50meters, i.e., the device may be at any location within 50 meters of thetransmitted position. The server may also assume a level of accuracy inthe absence of such information.

In another example, server 110 may extrapolate geographic locations fromone or more various types of information received from the user devices.For example, server 110 may receive location data from the user deviceindicating a potential location of the device. Location data may includean exact point, a set of points, a distance from a point, a range ofpoints, a set of points, or arbitrary boundaries, for example streets,cities, zip codes, counties, states or the like. Server 110 may alsodetermine a range of accuracy of the location data and use thisinformation to determine a position or area in which the user device maybe located.

Another location technology employs triangulation among multiple celltowers to estimate the device's position. A further location technologyis IP geocoding. In this technique, a client device's IP address on adata network may be mapped to a physical location. As noted before,locations may be provided in any number of forms including streetaddresses, points of interest, or GPS coordinates of the user device.

The detected geolocation data, elevation, tilt and orientation may betransmitted to a position and direction calculation module 412 forfurther processing and calculations. The calculation module may be a setof computer code that resides on the memory of the user device.

Various calculation techniques may be used to estimate the deviation ofthe user device to the traffic light. For example, a vector map may bebuilt for the area range approximately between the user's position tothe traffic light's position. In another example, a raster grid overlaymay be created where each cell in the grid overlay may be assigned anelevation data. Elevation data such as those of the user, of the trafficlight pole and of the other clutters within the area are included in themap. The maps may help the device to find a path of view from the deviceto the traffic light, so the device may calculate the direction andmagnitude with which the user device should be moved, and to furtherprovide the instructions prompting the user to move the device to thedesired position.

User device 160 may contain an image capture module 410 that comprises acamera 184, one or more sensors 180 and/or filters 182. This module maydetect traffic light data 404. The sensor 180 may comprise single ormulti element photo detector or monochrome/color detectors. The userdevice may also include a group of hardware bandpass filters 182 thatonly allow light at the desired wavelength to pass through. For example,the bandpass filters may only allow the emission frequencies of thetraffic light LEDs to go through.

The user device may further comprise an image processing module 414. Theimage processing module 414 may include one or more image processingprograms that perform the functions of image recognition and processing.For example, the camera 184 may capture a photo image of the trafficlight and the surrounding area. Various color and image processingmodels may be applied to process the captured image.

In one example, color filtering may be performed by an appropriatelytuned filter targeted for the red, yellow and/or green light(s). Soregions of red, yellow and green light indicating an active trafficlight may be identified in the image.

In another example, color screening may also be performed by convertingthe captured image of the traditional RGB (Red, Green, Blue) tri-colormodel into another representative model, such as the HIS (Hue,Saturation, and Intensity) color space, which is more related to humanperception and less sensitive to illumination variation. By applyingappropriate formulas or statistical models, HIS space may be used toscreen out the pixels not satisfying the threshold ranges. A binaryforeground map may thus be obtained by setting those pixels fallingwithin the desired range as foreground point with a positive value andthose pixels being screened out as background points having a value ofzero.

Various screening techniques may be applied to reduce the impact ofenvironmental variation, for example, by setting different illuminationconditions for daytime and nighttime. Because traffic lights are activelight sources emitting light in a particular direction, images may bepurposefully toned down or otherwise made darker to enhance the contrastbetween different a traffic light and other light sources.

Morphology technologies, such as erosion and dilation may be used tomask the foreground map and thus to remove noise and broken regions tofind the shape of the traffic light. Edge detection and shapecharacterization may also be performed to further process the image todetect the traffic lights.

In one scenario, there may be lights for different lanes on one trafficlight pole at an intersection. For example, as shown in FIG. 5A, trafficlight 102 at intersection 501 includes three traffic lights 102 a, 102 band 102 c with each respectively dedicated to one of the three lanes521-523. The green light of light 102 a for lane 521 may havealternating signs of left-turn arrow and straight arrow, while lights192-193 may bear no signs. Such references may be stored in a database,such as database 138, or in a server, such as server 110, containingtraffic light information. In this situation, the device may make use ofthe reference to the traffic light to decide which light the deviceshould detect, and prompt the user accordingly to point the device tothe light 102 c instead of the light 102 a. If no such reference isavailable, the device may take an image of the traffic light, andanalyze the image to find the appropriate traffic light.

In another scenario 600, as shown in FIG. 6, various types of pedestriantraffic lights may be present at an intersection. Some pedestrian lightsmay only perform pattern changing but not color changing. For example,the background color of pedestrian light 610 or 615 may stay the samewhen they change the signs from “DON'T WALK” to “WALK”, or from astanding person to a walking person. In these situations, the imageprocessing module 425 may include pattern and image recognition routinesto discern the signs on the traffic light. For example, classifiedmodels of traffic lights may be used by the image processing module 425for template matching. Existing classification and knowledge of thetraffic lights and signs may be obtained from a database on the networkor may be built into the image processing programs.

Many parameters may be taken into account to correctly identify thetraffic light status, e.g., geometrical status (vertical and horizontalorientation, size and position) of the traffic light, signs on thelight, timing sequence of the traffic light, and visual environmentalcures etc. The image processing routines are also robust to the visualclutters from other sources, e.g., neon light from a nearby building.Features other than visual characteristics, such as sound data, may alsobe used to determine the light status. For instance, audible speech maystate “walk” or “don't walk”; or chirping tones from a sound device 620in FIG. 6 (such as a transducer or speaker) may be incorporated into theconfiguration.

Alternatively, neural networks may be employed for recognition andclassification of the traffic lights and the related signs. In this way,large computations for template matching may be avoided. For example,the input image may not need to be transformed into anotherrepresentative space such as Hough space or Fourier domain. Therecognition result may depend only on the correlation between thenetwork weights and the network topology itself.

The user device 160 may include a prompt generating module 416 thatgenerates routines based on the deviation from the user device to thetraffic light. A speech synthesis module 418 may convert the promptsinto speech utterance and output the prompts to the user through thespeaker 168. The device may also provide the user with options to choosethe type of output, e.g., audio or tactile output.

It will be further understood that the sample values, types andconfigurations of data described and shown in the figures are for thepurposes of illustration only. In that regard, systems and methods inaccordance with aspects of the invention may include different trafficlight patterns, visual or audio characteristics, environmental features,data values, data types and configurations, and different image andsound processing techniques. The systems and methods may be provided andreceived at different times (e.g., via different servers or databases)and by different entities (e.g., some values may be pre-suggested orprovided from different sources).

As these and other variations and combinations of the features discussedabove can be utilized without departing from the invention as defined bythe claims, the foregoing description of exemplary embodiments should betaken by way of illustration rather than by way of limitation of theinvention as defined by the claims. It will also be understood that theprovision of examples of the invention (as well as clauses phrased as“such as,” “e.g.”, “including” and the like) should not be interpretedas limiting the invention to the specific examples; rather, the examplesare intended to illustrate only some of many possible aspects.

Unless expressly stated to the contrary, every feature in a givenembodiment, alternative or example may be used in any other embodiment,alternative or example herein. For instance, any technology fordetermining the location of a traffic light or a mobile device may beemployed in any configuration herein. Each way of communicating thelocation of a traffic light or the status of the light may be used inany configuration herein. Any mobile user device may be used with any ofthe configurations herein.

1. A method of identifying a traffic light status, where the statuscomprises at least one of a color illumination status and a sign statusof a traffic light, the method comprising: receiving, at a user device,geolocation data associated with the traffic light and the user device,the user device geolocation data including a location of the userdevice, and the traffic light geolocation data including a location ofthe traffic light; detecting an elevation and a direction of the userdevice; generating, with the user device, one or more instructions toorient the user device to face the traffic light based on thegeolocation data associated with the traffic light and the user device,and on the elevation, the direction and an orientation of the userdevice; capturing one or more images of the traffic light by an imagecapture device of the user device; performing with the user device imageprocessing on the captured images to determine a status of the trafficlight; and communicating the status to a user of the user device.
 2. Themethod of claim 1, wherein determining the status of the traffic lightwith the user device further comprising: determining from thegeolocation data of the user device if a current location is adjacent toan intersection; and prompting the user to orient the user device toface the traffic light if the current location is adjacent to theintersection.
 3. (canceled)
 4. The method of claim 1, wherein generatingthe one or more instructions to orient the user device furthercomprises: generating a map for an area between the user device and thetraffic light; and calculating deviations from the user device to thetraffic light based on the map.
 5. The method of claim 1, whereindetermining the status of the traffic light with the user device furthercomprises: receiving light signals from a plurality of light sources,where one of the light sources comprises the traffic light; andfiltering light signals for frequency ranges emitted by the trafficlight.
 6. (canceled)
 7. The method of claim 1, further comprising:receiving information related to the traffic light, the informationcomprising at least one of a size, a type and a timing sequence of thetraffic light; and wherein determining the status of the traffic lightwith the user device is performed based on the received informationrelated to the traffic light.
 8. The method of claim 1, whereincommunicating the traffic light status to the user further comprisesgenerating at least one of an audible output and a tactile output. 9.The method of claim 1, wherein determining the status of the trafficlight further comprises at least one of the recognizing signs associatedwith the traffic light, receiving audible information related to thestatus of the traffic light and determining the status of the trafficlight is further based on the audible information.
 10. The method ofclaim 1, wherein the location of the traffic light and the location ofthe user device are detected by a geographic position device, and thedirection associated with the user device is detected by a digitalcompass.
 11. A device for identifying a traffic light status of atraffic light, the device comprising: memory for storing geolocationdata of the traffic light, geolocation data of the device, and directioninformation and elevation information for the device; and a processorcoupled to the memory, the processor being configured to execute a setof instructions stored in the memory to: receive the geolocation data ofthe traffic light and the device; determine a current elevation and acurrent direction of the device; determine with the device one or moreinstructions to orient the device to face the traffic light based on thereceived geolocation data, the current elevation, the current directionand an orientation of the device; receive light signals from a pluralityof light sources, where one of the light sources comprises the trafficlight; filter light signals for frequency ranges emitted by the trafficlight, the status including at least one of a color illumination statusand a sign status; and communicate the status to a user of the device.12. The device of claim 11, wherein the processor is further configuredto determine the status of the traffic light by: determining, from thegeolocation data of the device, if a current location of the device isadjacent to an intersection; and prompting the user to orient the deviceto face the traffic light if the current location is adjacent to theintersection.
 13. (canceled)
 14. The device of claim 11, wherein theprocessor is further configured to generate the instructions by:generating a map for an area between the user device and the trafficlight; and calculating deviations from the device to the traffic lightbased on the map.
 15. (canceled)
 16. The device of claim 11, the devicefurther including an imager, and the processor is further configured to:capture one or more images of the traffic light and a surrounding areawith the imager; perform image processing on the captured images todetermine the status of the traffic light.
 17. The device of claim 11,wherein the processor is further configured to: receive informationrelated to the traffic light, including at least one of a size, a typeand a timing sequence of the traffic light, wherein the processordetermines the status of the traffic light using the receivedinformation related to the traffic light.
 18. The device of claim 11,wherein the processor is further configured to generate an audibleoutput or a tactile output of the traffic light status.
 19. The deviceof claim 11, wherein the processor determines the status of the trafficlight by at least one of recognizing signs associated with the trafficlight, and receiving audible information related to the status of thetraffic light.
 20. The device of claim 11, wherein the processordetermines the location of the traffic light and the location of thedevice using a geographic position device, and determines the directionof the device with a digital compass. 21-26. (canceled)